Method, apparatus, and system for blaming a test case/class for a survived mutation

ABSTRACT

A method and apparatus are provided for enhancing operable testing functionality for program code by mutating a code segment in a received program code to generate a mutated code segment, by monitoring execution of a test suite against the mutated code segment to detect a surviving mutation, and by evaluating the surviving mutation against the test suite to identify a first test case to blame for the surviving mutation by analyzing a coverage of the plurality of test cases in the test suite for the surviving mutation to identify one or more test cases from the test suite which cover the surviving mutation and then select the first test case which meets one or more predetermined selection criteria.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention is directed in general to field of informationprocessing. In one aspect, the present invention relates generally tosoftware testing.

Description of the Related Art

Computer programmers, developers, coders and/or software engineerswrite, test, debug, and maintain computer software or code instructions,called computer programs, which computers must follow to perform theirfunctions. When testing a computer program, a collection of test cases(collectively referred to as a test suite) is applied to test thecomputer program to show that it has some specified set of behaviors. Atest suite often contains detailed instructions or goals for eachcollection of test cases and information on the system configuration tobe used during testing. However, the quality of a test suite must beevaluated to determine not only the code coverage for the test suite(e.g., what percentage of the execution paths in the computer programhave been exercised during automated testing), but also the testaccuracy and user case completeness of the test suite. One approach forimproving the adequacy of tests to better identify defects in the codeis through mutation testing (a.k.a., mutation analysis or programmutation) whereby new software tests are designed through a processwhere the program code is modified or “mutated” in predetermined waysand then evaluated by a test suite to determine if the test suitedetects and rejects (or “kills”) the mutants. For example, mutations maybe generated by using well-defined mutation operators that either mimictypical programming errors (such as using the wrong operator or variablename) or force the creation of valuable tests (such as dividing eachexpression by zero). Test suites are measured by the percentage ofmutants that they kill, and new tests can be designed to kill anysurviving mutants. However, while conventional mutation test tools (suchas the PIT mutation testing system) can identify which mutationssurvived, they provide only block coverage reports, making it verydifficult for a developer/tester to understand which specific part ofthe test suite failed and should be modified to kill the survivingmutant(s). Additional drawbacks with conventional mutation test tools isthat they have significant rates of false positive reporting (e.g.,20-50%), require different plugins for different build systems (e.g.,pitest maven plugin, pitest ant plugin, etc.), have limited test failuretolerance, and/or limited stability (e.g., buggy). As seen from theforegoing, the existing solutions for performing mutation coveragetesting are extremely difficult at a practical level by virtue of thedifficulty in using the mutation test system to accurately andefficiently identify and upgrade test suites.

SUMMARY OF THE INVENTION

A system, apparatus, computer program product, and methodology aredescribed for efficiently identifying a test case and/or test class in atest suite that is responsible for a survived mutation to help a programdeveloper/tester to modify the old test case/class or write a new testcase in the correct test class to kill the survived mutation. Selectedembodiments of the present disclosure automatically apply specified testcase identification rules to a list of tests which cover a survivingmutation to identify one or more test cases or classes which are likelyresponsible for missing the mutation. After generating source codemutations and applying a test suite, each surviving mutation is analyzedto automatically identify which test case/method or test class isresponsible for missing the mutation. In a first step of the analysis,the process assembles a list (L) of all test case(s)/method(s) whichcover a given mutation, and if the list (L) contains a single testmethod/case, then the test method/case on the list is identified orblamed as being responsible for missing the mutation. In a second stepof the analysis, the process then filters the list (L) to remove anytest method(s)/case(s) that killed another mutation, and if theresulting filtered list contains a single test method/case, the thattest method/case is identified or blamed as being responsible formissing the mutation. In a third step of the analysis, the processidentifies a name (X) for the mutated method, and then searches throughthe filtered list to identify any test case/method which includes thename X as part of its name, in which case that matching test case/methodis identified or blamed as being responsible for missing the survivingmutation. And if no specific test method is identified, then theanalysis moves to the test class level by first determining if all ofthe test methods on the filtered list belong to a single test class, inwhich case the test class is identified/blamed as being responsible. Ifnot, then the process searches through a list (L2) of distinct testclass names for the test case(s)/method(s) remaining on the filteredlist to identify any class which includes the name X as part of itsname, in which case that matching test class is identified or blamed asbeing responsible for missing the surviving mutation. And if there is notest class which contains the name X, then the first test class on thetest class list (L2) is identified or blamed as being responsible formissing the surviving mutation.

The objects, advantages and other novel features of the presentinvention will be apparent from the following detailed description whenread in conjunction with the appended claims and attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be understood, and its numerous objects,features and advantages obtained, when the following detaileddescription of a preferred embodiment is considered in conjunction withthe following drawings.

FIG. 1 is a simplified block diagram of a data processing system foridentifying test cases responsible for surviving mutations and forgenerating new tests to address surviving mutations in accordance withselected embodiments of the present disclosure.

FIG. 2 illustrates a simplified flow chart showing the logic forautomatically identifying test cases responsible for surviving mutationsin accordance with selected embodiments of the present disclosure.

FIGS. 3A-C illustrate an example sequence of user interface screenswhich may be generated by the mutation testing engine that automaticallyidentifies test cases responsible for missing a surviving mutation inaccordance selected embodiments of the present disclosure.

FIG. 4 is a simplified block diagram of a general-purpose computer inaccordance with selected embodiments of the present disclosure.

DETAILED DESCRIPTION

A mutation testing engine, methodology, computer program product,system, and apparatus are described for efficiently improving test suitecoverage by automatically evaluating test cases which cover a survivingmutation against specified selection rules to identify a test case orclass which uniquely correspond to a surviving mutation for use inidentifying a test case that can be modified and/or write a new testcase in a correct test class to kill the survived mutation. Indisclosing the mutation testing engine, methodology, computer programproduct, system, and apparatus for blaming a test case or class by usingspecific rules, rather than humans, to provide an automated sequence foridentifying problematic test cases or classes with reports that provideline coverage for mapping mutations to responsible test cases, there isdisclosed an improvement in computer-related technology that uses acombined order of specific rules that renders information into aspecific format that is then used and applied to create desired resultsin the form of improved test suites for evaluating the performance ofcomputer programs.

While various details are set forth in the following description, itwill be appreciated that the present invention may be practiced withoutthese specific details. For example, selected aspects are shown in blockdiagram form, rather than in detail, in order to avoid obscuring thepresent invention. Some portions of the detailed descriptions providedherein are presented in terms of algorithms or operations on data withina computer memory. Such descriptions and representations are used bythose skilled in the data processing arts to describe and convey thesubstance of their work to others skilled in the art. In general, analgorithm refers to a self-consistent sequence of steps leading to adesired result, where a “step” refers to a manipulation of physicalquantities which may, though need not necessarily, take the form ofelectrical or magnetic signals capable of being stored, transferred,combined, compared, and otherwise manipulated. It is common usage torefer to these signals as bits, values, elements, symbols, characters,terms, numbers, or the like. These and similar terms may be associatedwith the appropriate physical quantities and are merely convenientlabels applied to these quantities. Unless specifically stated otherwiseas apparent from the following discussion, it is appreciated thatthroughout the description, discussions using terms such as processing,computing, calculating, determining, displaying or the like, refer tothe action and processes of a computer system, or similar electroniccomputing device, that manipulates and/or transforms data represented asphysical, electronic and/or magnetic quantities within the computersystem's registers and memories into other data similarly represented asphysical quantities within the computer system memories or registers orother such information storage, transmission or display devices.

Referring now to FIG. 1, a simplified block diagram illustrates anexemplary data processing system 100 for identifying test casesresponsible for surviving mutations and for generating new tests toaddress surviving mutations with one or more server/computer systems 110having processor(s) 111, memory 112, and associated database storagedevices 130 which may be connected together over a suitable connectionlink 120 or network, such as a private wide area network (WAN) or theInternet (not shown). To illustrate the operative functionality of thedata processing system 100, the server memory 112 may be used to storethe program module functionality for the mutation engine 113. Asdescribed hereinbelow, the mutation engine 113 acts upon the inputsource code files 131 and/or data from the database storage devices 130in a first processing step 121 to generate mutated source code files 134using any desired mutation technique for modifying the input source codefiles 131. For example, the mutation generator 114 may define aplurality of mutators, such as the Increments_Mutator,Void_Method_Call_Mutator, Return_Vals_Mutator, Math_Mutator,Negate_Conditionals_Mutator, Invert_Negs_Mutator, orConditionals_Boundary_Mutator. Using inputs from the mutation generator113, the source code modification module 115 may dynamically change oneor more application code segments 132, 133 in the input source codefiles 131, thereby generating mutated source code files 134 having oneor more mutated application code segments 135, 136 which are generatedfor the purpose of causing test failures in the mutated source code 134.

To detect such failures, the server/computer system 110 may also includea test engine 117 that is configured to test the mutated source codefiles 134-136 by applying one or more test suites 137. For example, thetest engine 117 may compile and build the modified source code 134 andrun one or more test generator modules 118 to generate one or more testsuites 137 that may be stored in the database 130, and then run the testsuite(s) 137 on the mutated source code files 134 (processing step 122)to identify any survived mutations 138 (processing step 123). If one ofthe tests in the test suite 137 “fails” upon detecting a change causedby a mutated source code file (e.g., 135), then the mutant is said to bekilled. However, if the test suite 137 does not detect a change causedby a mutated source code file (e.g., no test fails), then the mutatedsource code file is said to “survive,” and may be collected with thesurvived mutations 138 (processing step 123).

As disclosed herein, the server/computer system 110 may include computerprogram functionality for identifying a specific test case and/or testclass in the test suite 137 that is likely responsible for missing thesurvived mutation. To this end, the mutation engine 113 may also includea test blame module 116 which is configured to identify or “blame” atest case or test class 139 (processing step 124) help a programdeveloper/tester understand which test should be changed to make themutation die. As described more fully hereinbelow, the test blame module116 acts upon each survived mutation 138 to identify a list (L) of alltest cases from the test suite 137 which cover the survived mutation.For example, the test suite 137 of all test cases for the input sourcecode 131 may be run to get the line coverage details for each sourcecode file using any suitable code coverage library tool, such as JaCoCo,Sonar, EclEmma, Cobertura, CodeCover, Coverage.py, Gretel, JCov, and thelike. Then, the list of covering test cases is evaluated against aseries of rules or conditions that may be used to blame either a testcase or test class that is responsible for missing the survivedmutation. Generally speaking the narrowing down the list of coveringtest cases through a combination of filtering test cases which killedanother mutation and name matching techniques until a single remainingor matching test case or class can be identified or blamed. With anassembled listing of blamed test cases/classes 139, the programdeveloper/tester can generate new tests 140 (processing step 125), suchas by modifying the blamed test case/class or writing a new test case inthe blamed test class to kill the survived mutation. By updating thetest suite 137 with the new test(s) 140 (processing step 126), the testengine 117 can run more effective test suites against the input sourcecode 131 and/or mutated source code 134.

The operation of the mutation engine 113 and/or test engine 117 totransform input source code files 131 into mutated source code files 134which are then tested with the test suites 137 to identify and blametest cases 139 that are responsible for survived mutations 138 throughthe automated action of the source code modification module 116 is shownwith processing steps 121-126, though it will be appreciated that theunderlying data may be stored in the database storage device 130 and/ormemory 112.

In selected illustrative embodiments, the server/computer system 110 maybe embodied as a Question Answering (QA) system to use NLP, machinelearning, and/or artificial intelligence processing techniques which areaugmented with the mechanisms of the illustrative embodiments describedhereafter. In a QA system implementation, the mutation engine 113 and/ortest engine 117 may be embodied as a computer program stored in memory112 which uses one or more processors 111 to query a structured orunstructured knowledge or information database 130. In particular, themutation engine 113 may retrieve one or more input source code files131-133 from the database 130 for use in generating mutated source codefiles 134. In addition, the test engine 117 may retrieve the mutatedsource code files 134 and one or more test suites 137 which aregenerated and/or stored, and then run the test suite(s) 137 against themutated source code files 134. Using the test blame module 116, themutation engine 113 may then identify survived mutations 138 forevaluation against specified selection rules to identify or “blame” thetest case or class 139 which is responsible for each survived mutation.The configuring of the mutation and testing engines 113, 117 may includeproviding application specific hardware, firmware, and/or software toimplement one or more control logic modules 113-118 or the like tofacilitate the performance of the operations and generation of theoutputs described herein with regard to the illustrative embodiments. Inaddition or in the alternative, the configuring of the mutation andtesting engines 113, 117 may include storing software applications inone or more storage devices and loaded into memory of a computingdevice, such as server 110, for causing one or more hardware processors(e.g., 111) of the computing device to execute the software applicationsthat configure the processors to perform the operations and generate theoutputs described herein with regard to the illustrative embodiments.Moreover, any combination of application specific hardware, firmware,software applications executed on hardware, or the like, may be usedwithout departing from the spirit and scope of the illustrativeembodiments.

In selected embodiments, test cases or classes that are “blamed” formissing a mutation may be presented to the program/test developer in auser interface display screen which has a first mutation pane (providingsummary information about a surviving mutation that was not caught bythe test suite), a second test case pane (providing summary informationabout the test case which is blamed for not killing the mutation), andone or more additional code panes (providing summary information for theoriginal code and/or mutated code). The user interface display screenmay also include a link for viewing the original code and/or injectedmutation.

As will be appreciated, once the server/computer system 110 isconfigured to implement the mutation and testing engines 113, 117, theserver/computer system 110 becomes a specialized computing devicespecifically configured to implement the mechanisms of the illustrativeembodiments and is not a general purpose computing device. Moreover, asdescribed hereafter, the implementation of the mechanisms of theillustrative embodiments improves the functionality of the computingdevice and provides a useful and concrete result that facilitates theautomatic identification or “blaming” of test cases and classes from thetest suite which cover the mutation and are likely responsible for notkilling the mutation, thereby identifying a candidate test case for asurvived mutation which the developer/tester may modify or write a newtest case in the correct test class to kill the survived mutation.

To provide a contextual understanding for selected embodiments of thepresent disclosure, reference is now made to FIG. 2 which illustrates asimplified mutation test processing workflow 200 showing the logicand/or processing steps for automatically identifying test casesresponsible for surviving mutations in accordance selected embodimentsof the present disclosure. In selected embodiments, the depictedmutation test processing workflow 200 may be implemented in whole or inpart with a data processing system (such as shown in FIG. 1) and/or asingle integrated circuit (e.g., a system-on-chip) or a plurality ofintegrated circuits to implement one or more processors with localmemory and database storage that are interconnected and interoperateusing any suitable technique without departing from the presentinvention.

However implemented, the mutation test processing workflow 200 receivesinformation identifying one or more survived mutations (step 201). Aswill be appreciated, a list of survived mutations may be received uponperforming mutation testing or analysis on input source code having oneor more mutated files (e.g., App Code A′ or App Code B′ in FIG. 1) whichmay be created by a source code modifier which applies mutationoperators to an input source code file. On the modified source code,test cases from a test suite are run to identify surviving mutationswhich are not killed by the test suite, and the process then evaluateseach mutation in an attempt to blame a test case or class that isresponsible for missing the mutation.

As a preliminary step 202, each surviving source code mutation isevaluated to identify and collect a list of all of the test cases ormethods from the test suite which cover the survived mutation. As willbe appreciated, the processing step 202 may be performed by the mutationengine 113 or the blame module 116 using any suitable coverage frameworkwhich provides line coverage details for each surviving source codemutation, including but not limited to the JaCoCo code coverage librarytool or similar such tools. To provide an example case for purposes ofillustrating this step, a survived mutation received at step 201 is themethod, “A.B.check( ),” and the processing step 201 identifies the testcases which cover this mutation as “A.B.testCheck( )” and“A.B.testBreak( ).” As a result of step 202, a list “L” of covering testmethods or cases is generated. As a result of step 202, a list “L” ofcovering test methods or cases is generated.

After assembling the list (L) of covering test methods, the list isevaluated at step 203 to see how many test methods are on the list. Ifan affirmative outcome to detection step 203 indicates that only asingle test method covers the surviving mutation (e.g., L=1), then thesingle test method is blamed for the surviving mutation (step 204).However, a negative outcome to detection step 203 indicates that thereare multiple covering test methods for the surviving mutation (e.g.,L>1), in which case additional processing is required. This isillustrated with the example case where the processing at step 203determines that there are two covering test cases (namely,“A.B.testCheck( )” and “A.B.testBreak( )”), so the process continues tostep 205.

At step 205, the list of covering test methods is filtered to remove anytest methods that also killed another mutation. In selected embodiments,the processing step 205 may be performed by the mutation engine 113 orthe blame module 116 using any suitable filtering technique, such asmarking as “ignored” any test cases which killed at least one mutation,and then deleting the ignored tests from the list over covering testcases. After filtering the assembled list (L) of covering test methods,the filtered list is evaluated at step 206 to see how many test methodsremain. If an affirmative outcome to detection step 206 indicates thatonly a single filtered test method covers the surviving mutation (e.g.,L=1), then the single test method is blamed for the surviving mutation(step 204). However, a negative outcome to detection step 206 indicatesthat there are still multiple covering test methods for the survivingmutation (e.g., L>1), in which case additional processing is required.This is illustrated with the example case where the processing at step206 determines that the list (L) contains multiple covering test methodssince neither of the covered test methods (namely, “A.B.testCheck( )”and “A.B.testBreak( )”) were filtered.

At step 207, another test method identification process begins whichseeks to use name matching techniques to identify a test case whichmatches the surviving mutation. In particular, the processing step 207may use the mutation engine 113 or the test blame module 116 to identifya name (e.g., “X”) of the mutated source code. In selected embodiments,the processing step 207 uses any suitable extraction technique, such asnatural language processing (NLP) techniques to find a method name forthe source code where the mutation happened. This is illustrated withthe example case where the processing at step 207 identifies the namestring “X”=“check( )” from the method name where the mutation happened.After determining the method name, the name is evaluated at step 208 tofind a match with any covering test methods in the list. If anaffirmative outcome to detection step 208 indicates that the name “X” isa substring in one of the covering test methods, then the matching testmethod (or methods) is blamed for the surviving mutation (step 204).This is illustrated with the example case where the processing at step208 determines that there is a test case (e.g., “A.B.testCheck( )”)having the method name “check( )” in it, so the matching test case(e.g., “A.B.testCheck( )”) is blamed at step 204. As will beappreciated, if multiple test methods match the name component, theneach test method may be blamed at step 204. However, if no match isdetected (e.g., negative outcome to detection step 208), then thisindicates that additional processing is required.

At this point in the “blaming” process sequence 201-208, no single testcase in the list (L) has been blamed since the filtering and namematching techniques have failed to sufficiently narrow the list (L) to asingle culprit. Accordingly, the next stage in the process sequence isto check at step 209 if all the test cases in the list (L) belong to thesame test class which can be blamed. If the all of the test cases in thefiltered list belong to a single test class (affirmative outcome todetection step 209), then the test class is blamed (step 210), in whichcase the program developer/tester can be directed to write a new testcase in the correct test class to kill the survived mutation. However,if the covered test methods in the filtered list belong to multiple testclasses (negative outcome to detection step 209), then this indicatesthat additional processing is required.

Beginning with step 211, the “blaming” process begins applying namematching techniques to identify a test class from the list of test caseswhich matches the surviving mutation. In particular, the processing step211 may use the mutation engine 113 or the test blame module 116 toapply any suitable extraction technique, such as natural languageprocessing (NLP) techniques, to find distinct test class names for alltest methods in the filtered list (L). Once the test class names areassembled in a second list (L2), the test class names are evaluatedagainst the previously identified name (e.g., “X”) of the mutated sourcecode to find a match at step 212. If an affirmative outcome to detectionstep 212 indicates that the name “X” is a substring in one of thedistinct test class names in the second list (L2), then the matchingtest class (or test classes) is blamed for the surviving mutation (step210). As will be appreciated, if multiple test classes match the namecomponent, then each test class may be blamed at step 210. However, ifno match is detected (e.g., negative outcome to detection step 212),then the first test class in the second list (L2) is blamed (step 213).As will be appreciated, the rank ordering of the test classes in thesecond list (L2) may be determined based on relative weighting orscoring factors, such as the number of lines covered by each test class,but any suitable ordering scheme may be used to rank the test clases.

Once a test method or class is identified or “blamed” as beingresponsible for a survived mutation (steps 210, 213), the test suite maybe modified by the program/test developer in to either modify the“blamed” test case or write a new test case in the “blamed” test classto kill the survived mutation. To this end, the identified test cases orclasses that are “blamed” for missing a mutation may be presented to theprogram/test developer in a user interface display screen.

To provide additional details for an improved understanding of selectedembodiments of the present disclosure, reference is now made to FIGS.3A-C which depict an example sequence of user interface screen shotswhich may be generated by the mutation testing engine that automaticallyidentifies test cases responsible for missing a surviving mutation inaccordance selected embodiments of the present disclosure. In the firstexample screen shot of user interface 300 shown in FIG. 3A, there isdisplayed a first user interface screen 300 which shows the “blamed”test case along with information identifying the mutation that happened.For example, the user interface 300 includes information identifying thetest case that is blamed for a surviving mutation that was not killed bythe test suite, such as the name of the test case(com.devfactory.ui.backend.service.StatusServiceTest.shouldReturnEmptyForNonExistentFile)in the title bar or a first test case pane 302 which provides summaryinformation about the test case from the test suite case which is blamedfor not killing the mutation surviving. The user first user interfacescreen 300 may also include a second mutation pane 303 which providessummary information about the surviving mutation, such as informationidentifying the method (e.g.,com.devfactory.aline.base.testing.VersionedS3Data) and the type ofmutation (e.g., Return Values Mutator). In selected embodiments, thefirst user interface screen 300 may include one or more additionalinformation panes, such as a first pane 304 which lists the inputbytecode before mutation and a second pane 305 which lists the inputbytecode after mutation, including one or more visual indications 306,307 to specify what mutation was actually made. If the user wants to seethe original source code, the first user interface screen 300 mayinclude a first button 308 or other user-activated link for opening anew user interface screen in which the input source code can be seenwith a more detailed view showing the specific code lines where themutations were made.

By using the cursor or other user interface controls to interact withthe first user interface screen 300, the developer may cause themutation testing engine to display a second user interface screen shot310 which displays the original source code (as indicated by theselected button 311), as illustrated in FIG. 3B. In this example, theuser interface 310 provides a button 311 which is selected to displaythe original source code in a source code viewing pane 312 wherein theoriginal source code 313 is displayed with corresponding line numbersand a visual indication 314 specifying the original source code prior tomutation.

By using the cursor or other user interface controls to interact withthe second user interface screen 310, the developer may cause themutation testing engine to display a third user interface screen shot320 which also displays the mutated source code (as indicated by theselected button 321), as illustrated in FIG. 3C. In this example, theuser interface 320 includes a source code viewing pane 322 whichdisplays, not only the original source code 313 and associated visualindication 314, but also displays the mutated source code 323 withcorresponding line numbers and a visual indication 324 specifying theinjected mutation. With the third user interface screen shot 320, themutation testing engine provides a way for the program/test developer tosee what change actually happened at the bytecode level by using adecompiler to get it in the code form again to show these as source codeto help the developer to easily understand the change.

Embodiments of the system and method for identifying test cases orclasses that are responsible for missing mutations can be implemented ona computer system, such as a general-purpose computer 400 illustrated inFIG. 4. As disclosed the computer 400 includes input user device(s) 416,such as a keyboard and/or mouse, which are coupled to a bi-directionalsystem bus 408. The input user device(s) 416 are used for introducinguser input to the computer system 400 and communicating that user inputto processor 402. The computer system 400 may also include a videomemory 404, main memory 406, and mass storage 418, all coupled tobi-directional system bus 408 along with input user device(s) 416 andprocessor 402. The mass storage 418 may include both fixed and removablemedia, such as other available mass storage technology. Bus 408 maycontain, for example, 32 address lines for addressing video memory 404or main memory 406. The system bus 408 may also include, for example, ann-bit data bus for transferring data between and among the components,such as CPU 402, main memory 406, video memory 414, and mass storage418, where “n” is, for example, 32 or 64. Alternatively, multiplexdata/address lines may be used instead of separate data and addresslines.

The computer 400 may also include I/O device(s) 410 which provideconnections to peripheral devices, such as a printer, and may alsoprovide a direct connection to remote server computer systems via atelephone link or to the Internet via an ISP. I/O device(s) 410 may alsoinclude a network interface device to provide a direct connection toremote server computer systems via a direct network link to the Internetvia a POP (point of presence). Such connection may be made using, forexample, wireless techniques, including digital cellular telephoneconnection, Cellular Digital Packet Data (CDPD) connection, digitalsatellite data connection or the like. Examples of I/O devices includemodems, sound and video devices, and specialized communication devicessuch as the aforementioned network interface.

Computer programs and data are generally stored as instructions and datain mass storage 418 until loaded into main memory 406 for execution.Computer programs may also be in the form of electronic signalsmodulated in accordance with the computer program and data communicationtechnology when transferred via a network. The method and functionsrelating to system and method for automatically blaming testcases/classes may be implemented in a computer program for a mutationtest engine 401 which evaluates a list of survived mutations 403 againsta plurality of covering test cases for each survived mutation toidentify or blame a corresponding test case or class 405 for eachsurvived mutation.

The processor 402, in one embodiment, is a microprocessor manufacturedby Motorola Inc. of Illinois, Intel Corporation of California, orAdvanced Micro Devices of California. However, any other suitable singleor multiple microprocessors or microcomputers may be utilized. Mainmemory 406 is comprised of dynamic random access memory (DRAM). Videomemory 404 is a dual-ported video random access memory. One port of thevideo memory 404 is coupled to video amplifier or driver 412. The videoamplifier 412 is used to drive the display 414. Video amplifier 412 iswell known in the art and may be implemented by any suitable means. Thiscircuitry converts pixel data stored in video memory 404 to a rastersignal suitable for use by display 414. Display 414 is a type of monitorsuitable for displaying graphic images.

By now, it will be appreciated that there is disclosed herein a system,method, apparatus, and computer program product for enhancing operabletesting functionality for program code by performing a method at adevice having a processor and a memory. As disclosed, the system,method, apparatus, and computer program receive a program and associatedtest suite comprising a plurality of test cases submitted by adeveloper, such as by compiling source code into program code.Subsequently, a mutation is performed on a code segment in the programcode to generate a mutated code segment, thereby transforming theprogram code into mutated program code using one or more predeterminedmutation rules. In selected embodiments, the mutation of the codesegment is performed by modifying one or more values within the programcode. After generating the mutated code, execution of the test suiteagainst the mutated code segment is monitored to detect a survivingmutation which is evaluated against the test suite to identify a firsttest case to blame for the surviving mutation by analyzing a coverage ofthe plurality of test cases in the test suite for the surviving mutationto identify one or more test cases from the test suite which cover thesurviving mutation and then select the first test case which meets oneor more predetermined selection criteria. In selected embodiments, thesurviving mutation is evaluated against the test suite by determiningthat the one or more test cases include only a single test case which isselected as the first test case under a first predetermined selectioncriteria. In other embodiments, the surviving mutation is evaluatedagainst the test suite by filtering the one or more test cases to removeany test case which killed another mutation so that only a single testcase remains which is selected as the first test case under a secondpredetermined selection criteria. In other embodiments, the survivingmutation is evaluated against the test suite by comparing a name segmentfrom the surviving mutation with name segments from the one or more testcases to identify a single test case having a matching name segmentwhich is selected as the first test case under a third predeterminedselection criteria. In addition, the surviving mutation may be evaluatedagainst the test suite to identify a first test class to blame for thesurviving mutation by assembling a list of distinct test classes fromthe one or more test cases in the test suite which cover the survivingmutation and then selecting the first test class which meets one or moretest class selection criteria. In selected embodiments, the survivingmutation is evaluated to identify the first test class by determiningthat the list of distinct test classes includes only a single test classwhich is selected as the first test class under the one or more testclass selection criteria. In other embodiments, the, the survivingmutation is evaluated to identify the first test class by comparing aname segment from the surviving mutation with name segments from thelist of distinct test classes identify a single test class having amatching name segment which is selected as the first test class. Once afirst test case or test class is selected, a recommendation may bepresented that the first test case or test class be modified to detectthe surviving mutation, where the recommendation comprises one or moreuser interface screens displaying information identifying the first testcase and the surviving mutation with line coverage information for thesurvived mutation.

The present invention may be a system, a method, and/or a computerprogram product such that selected embodiments include software thatperforms certain tasks. The software discussed herein may includescript, batch, or other executable files. The software may be stored ona machine-readable or computer-readable storage medium, and is otherwiseavailable to direct the operation of the computer system as describedherein and claimed below. In one embodiment, the software uses a localor database memory to implement the data transformation and datastructures so as to automatically detect and identify test cases orclasses that are likely responsible for each survived mutation, therebyimproving the quality and robustness of software, educating developersabout test suite upgrade opportunities and implementations, and generatemore readable, reliable, smaller, and robust code with less effort. Thelocal or database memory used for storing firmware or hardware modulesin accordance with selected embodiments of the present disclosure mayalso include a semiconductor-based memory, which may be permanently,removably or remotely coupled to a microprocessor system. Other new andvarious types of computer-readable storage media may be used to storethe modules discussed herein. Additionally, those skilled in the artwill recognize that the separation of functionality into modules is forillustrative purposes. Alternative embodiments may merge thefunctionality of multiple software modules into a single module or mayimpose an alternate decomposition of functionality of modules. Forexample, a software module for calling sub-modules may be decomposed sothat each sub-module performs its function and passes control directlyto another sub-module.

In addition, selected aspects of the present disclosure may take theform of an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.), or anembodiment combining software and/or hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present disclosure may take the form ofcomputer program product embodied in a computer readable storage mediumor media having computer readable program instructions thereon forcausing a processor to carry out aspects of the present disclosure. Thusembodied, the disclosed system, a method, and/or a computer programproduct is operative to improve the design, functionality andperformance of software programs by automatically detecting andrecommending “blamed” test cases or classes in the test suite forupgrade or replacement to better detect mutations.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a dynamic or static random access memory(RAM), a read-only memory (ROM), an erasable programmable read-onlymemory (EPROM or Flash memory), a magnetic storage device, a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a Public SwitchedCircuit Network (PSTN), a packet-based network, a personal area network(PAN), a local area network (LAN), a wide area network (WAN), a wirelessnetwork, or any suitable combination thereof. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present disclosure may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, or either source code or object code written in anycombination of one or more programming languages, including an objectoriented programming language such as Java, Python, Visual Basic.net,Ruby, Smalltalk, C++ or the like, and conventional proceduralprogramming languages, such as the “C” programming language, HypertextPrecursor (PHP), or similar programming languages. The computer readableprogram instructions may execute entirely on the user's computer, partlyon the user's computer, as a stand-alone software package, partly on theuser's computer and partly on a remote computer or entirely on theremote computer or server or cluster of servers. In the latter scenario,the remote computer may be connected to the user's computer through anytype of network, including a local area network (LAN) or a wide areanetwork (WAN), or the connection may be made to an external computer(for example, through the Internet using an Internet Service Provider).In some embodiments, electronic circuitry including, for example,programmable logic circuitry, field-programmable gate arrays (FPGA), orprogrammable logic arrays (PLA) may execute the computer readableprogram instructions by utilizing state information of the computerreadable program instructions to personalize the electronic circuitry,in order to perform aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a sub-system, module, segment,or portion of instructions, which comprises one or more executableinstructions for implementing the specified logical function(s). In somealternative implementations, the functions noted in the block may occurout of the order noted in the figures. For example, two blocks shown insuccession may, in fact, be executed substantially concurrently, or theblocks may sometimes be executed in the reverse order, depending uponthe functionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

The computer system described above is for purposes of example only, andmay be implemented in any type of computer system or programming orprocessing environment, or in a computer program, alone or inconjunction with hardware. Various embodiments of the present may alsobe implemented in software stored on a computer-readable medium andexecuted as a computer program on a general purpose or special purposecomputer. For clarity, only those aspects of the system germane to theinvention are described, and product details well known in the art areomitted. For the same reason, the computer hardware is not described infurther detail. It should thus be understood that the invention is notlimited to any specific computer language, program, or computer. Thesystem and method for identifying and recommending test suite upgradesmay be implemented in any type of computer system or programming orprocessing environment. It is contemplated that the system and methodfor generating test suite upgrade recommendations for addressingsurvived mutations might be run on a stand-alone computer system, suchas the one described above. The system and method for generating testsuite upgrade recommendations might also be run from a server computersystems system that can be accessed by a plurality of client computersystems interconnected over an intranet network. Finally, the system andmethod for generating test suite upgrade recommendations may be run froma server computer system that is accessible to clients over theInternet.

Benefits, other advantages, and solutions to problems have beendescribed above with regard to specific embodiments. However, thebenefits, advantages, solutions to problems, and any element(s) that maycause any benefit, advantage, or solution to occur or become morepronounced are not to be construed as a critical, required, or essentialfeature or element of any or all the claims. As used herein, the terms“comprises,” “comprising,” or any other variation thereof, are intendedto cover a non-exclusive inclusion, such that a process, method,article, or apparatus that comprises a list of elements does not includeonly those elements but may include other elements not expressly listedor inherent to such process, method, article, or apparatus.

What is claimed is:
 1. A method performed by a device having a processorand memory for enhancing operable testing functionality for programcode, comprising: receiving, by the device, a program code andassociated test suite comprising a plurality of test cases; mutating, bythe device, a code segment in the program code to generate a mutatedcode segment; monitoring, by the device, execution of the test suiteagainst the mutated code segment to detect a surviving mutation; andevaluating, by the device, the surviving mutation against the test suiteto identify a first test case to blame for the surviving mutation byanalyzing a coverage of the plurality of test cases in the test suitefor the surviving mutation to identify one or more test cases from thetest suite which cover the surviving mutation and then select the firsttest case which meets one or more predetermined selection criteria,wherein evaluating the surviving mutation against the test suitecomprises at least one of (1) determining that the one or more testcases include only a single test case which is selected as the firsttest case under a first predetermined selection criteria, (2) filteringthe one or more test cases to remove any test case which killed anothermutation so that only a single test case remains which is selected asthe first test case under a second predetermined selection criteria, or(3) comparing a name segment from the surviving mutation with namesegments from the one or more test cases to identify a single test casehaving a matching name segment which is selected as the first test caseunder a third predetermined selection criteria.
 2. The method of claim1, where receiving the program code comprises compiling source code intoprogram code.
 3. The method of claim 1, where mutating the code segment,comprises modifying one or more values within the program code.
 4. Themethod of claim 1, further comprising evaluating, by the device, thesurviving mutation against the test suite to identify a first test classto blame for the surviving mutation by assembling a list of distincttest classes from the one or more test cases in the test suite whichcover the surviving mutation and then selecting the first test classwhich meets one or more test class selection criteria.
 5. The method ofclaim 4, where evaluating the surviving mutation against the test suiteto identify the first test class comprises determining that the list ofdistinct test classes includes only a single test class which isselected as the first test class under the one or more test classselection criteria.
 6. The method of claim 1, further comprisingpresenting, by the device, a recommendation that the first test case bemodified to detect the surviving mutation, where the recommendationcomprises one or more user interface screens displaying informationidentifying the first test case and the surviving mutation with linecoverage information for the survived mutation.
 7. A computer programproduct comprising at least one computer readable storage medium havingstored thereon executable instructions and data which, when executed byat least one processing device, cause the at least one processing deviceto enhance operable testing functionality for a program code by:receiving the program code and associated test suite comprising aplurality of test cases; mutating a code segment in the program code togenerate a mutated code segment, thereby transforming the program codeinto mutated program code; monitoring execution of the test suiteagainst the mutated code segment to detect a surviving mutation; andevaluating the surviving mutation against the test suite to identify afirst test case to blame for the surviving mutation by analyzing acoverage of the plurality of test cases in the test suite for thesurviving mutation to identify one or more test cases from the testsuite which cover the surviving mutation and then select the first testcase which meets one or more predetermined selection criteria, whereinevaluating the surviving mutation against the test suite comprises atleast one of (1) determining that the one or more test cases includeonly a single test case which is selected as the first test case under afirst predetermined selection criteria, (2) filtering the one or moretest eases to remove any test case which killed another mutation so thatonly a single test case remains which is selected as the first, testcase under a second predetermined selection criteria, or (3) comparing aname segment from the surviving mutation with name segments from the oneor more test cases to identify a single test case having a matching namesegment winch is selected as the first test case under a thirdpredetermined selection criteria.
 8. The computer program product ofclaim 7, wherein the executable instructions and the data, when executedby the at least one processing device, causes the at least oneprocessing device to receive the program code by compiling source codeinto program code.
 9. The computer program product of claim 7, whereinthe executable instructions and the data, when executed by the at leastone processing device, causes the at least one processing device tomutate the code segment by modifying one or more values within theprogram code.
 10. The computer program product of claim 7, wherein theexecutable instructions and the data, when executed by the at least oneprocessing device, causes the at least one processing device to evaluatethe surviving mutation against the test suite to identify a first testclass to blame for the surviving mutation by assembling a list ofdistinct test classes from the one or more test cases in the test suitewhich cover the surviving mutation and then selecting the first testclass which meets one or more test class selection criteria.
 11. Thecomputer program product of claim 10, wherein the executableinstructions and the data, when executed by the at least one processingdevice, causes the at least one processing device to evaluate thesurviving mutation against the test suite to identify the first testclass by determining that the list of distinct test classes includesonly a single test class which is selected as the first test class underthe one or more test class selection criteria.
 12. The computer programproduct of claim 7, wherein the executable instructions and the data,when executed by the at least one processing device, causes the at leastone processing device to present a recommendation that the first testcase be modified to detect the surviving mutation, where therecommendation comprises one or more user interface screens displayinginformation identifying the first test case and the surviving mutationwith line coverage information for the survived mutation.
 13. A systemcomprising: one or more processors; a memory coupled to at least one ofthe processors; and a set of instructions stored in the memory andexecuted by at least one of the processors to enhance operable testingfunctionality for program code, wherein the set of instructions areexecutable to perform actions of: receiving a program code andassociated test suite comprising a plurality of test cases, mutating acode segment in the program code to generate a mutated code segment,thereby transforming the program code into mutated program code,monitoring execution of the test suite against the mutated code segmentto detect a surviving mutation; and evaluating the surviving mutationagainst the test suite to identify a first test ease to blame for thesurviving mutation by analyzing a coverage of the plurality of testcases in the test, suite for the surviving mutation to identify one ormore test cases from the test suite which cover the surviving mutationand then select the first test case which meets one or morepredetermined selection criteria, wherein evaluating the survivingmutation against the test suite comprise at least one of (1) determiningthat the one or more test cases include only a single test case which isselected as the first test case under a first predetermined selectioncriteria, (2) filtering the one or more test cases to remove any testcase which killed another mutation so that only a single test caseremains which is selected as the first test case under a secondpredetermined selection criteria, or (3) comparing a name segment fromthe surviving mutation with name segments from the one or more testcases to identify a single test case having a matching name segmentwhich is selected as the first test case under a third predeterminedselection criteria.
 14. The system of claim 13, further comprisingevaluating the surviving mutation against the test suite to identify afirst test class to blame for the surviving mutation by assembling alist of distinct test classes from the one or more test cases in thetest suite which cover the surviving mutation and then selecting thefirst test class which meets one or more test class selection criteria.15. The system of claim 14, where evaluating the surviving mutationagainst the test suite to identify the first test class comprisesdetermining that the list of distinct test classes includes only asingle test class which is selected as the first test class under theone or more test class selection criteria.